Project #03 Multimodal Guitar: Performance Toolbox and Study Workbench

This project aims at studying how recent interactive and interaction technologies would help extend how we play the guitar, thus defining the “multimodal guitar”. We investigate two axes, 1) “A gestural/polyphonic sensing/processing toolbox to augment guitar performances”, and 2) “An interactive guitar score following environment for adaptive learning”. These approaches share quite similar technological challenges (sensing, analysis, processing, synthesis and interaction methods) and dissemination intentions (community-based, low-cost, open-source whenever possible), while leading to different applications (respectively artistic and educational), still targeted towards experienced players and beginners. We designed and developed a toolbox for multimodal guitar performances containing the following tools: Polyphonic Pitch Estimation (see section III-A1), Fretboard Grouping (see section III-A2), Rear-mounted Pressure Sensors (see section III-B), Infinite Sustain (see section III-C2), Rearranging Looper (see section III-C3), Smart Harmonizer (see section III-C4). The Modal Synthesis tool (see section III-C1) needs be refined before being released. We designed a low-cost offline system for guitar score following (see section IV-A). An audio modality, polyphonic pitch estimation from a monophonic audio signal, is the main source of the information (see section IV-B), while the visual input modality, finger and headstock tracking using computer vision techniques on two webcams, provides the complementary information (see section IV-C). We built a stable data acquisition approach towards low information loss (see section IV-E). We built a probability-based fusion scheme so as to handle missing data; and unexpected or misinterpreted results from single modalities so as to have better multi-pitch transcription results (see section IV-D). We designed a visual output modality so as to visualize simultaneously the guitar score and feedback from the score following evaluation (see section IV-F). The audio modality and parts of the visual input modality are already designed to run in realtime, we need to improve the multimodal fusion and visualization so that the whole system can run in realtime.